Skip to content

Support multiple actors on the same rank #494

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

samlurye
Copy link

Summary: Previously, debug sessions were keyed by rank, but this doesn't allow for multiple actors on the same rank. This diff updates the debugging logic so that it is keyed on actor name + rank instead of just rank. The attach and cast commands now require the user to specify actor name in addition to rank. The list command now sorts breakpoints by grouping all actors with the same name (i.e., from the same actor mesh) together, and then sorts by rank within each group.

Differential Revision: D78047960

Summary:
Pull Request resolved: pytorch-labs#456

This diff contains several updates to the monarch actor mesh debugging experience.
- Improved debug input parsing, with the new `cast` command supporting more sophisticated rank selection grammar:
  - `cast ranks(3) pdb_command`: send `pdb_command` to rank 3.
  - `cast ranks(1,3,5) pdb_command`: send `pdb_command` to ranks 1, 3 and 5.
  - `cast ranks(1:10:2) pdb_command`: send `pdb_command` to the ranks in `range(start=1, stop=10, step=2)`.
  - `cast ranks(pp=2, dp=(1,3), tp=2:8) pdb_command`: send `pdb_command` to ranks with `pp` dim 2, `dp` dim 1 or 3, and `tp` dim in `range(2,8)`.
- The debug client is now automatically registered with an actor mesh when that actor mesh is spawned. This means calling `init_debugging(actor_mesh)` is no longer necessary.
- Debugging now works with MAST jobs, by enforcing that breakpoints aren't set in `__main__`, and that the file containing the breakpoint exists on the remote host.
  - The first requirement is due to how `cloudpickle` works -- if an actor endpoint is defined inside `__main__`, `cloudpickle` will serialize it by value instead of by reference. When the code then runs on the remote host, it thinks the location of the code is the user's local `__main__` file, which confuses pdb, because the file doesn't exist at the same path (or may not exist at all) on the remote host.
  - The second requirement is due to important parts of `pdb`'s implementation relying on the ability to search for the file being debugged on the remote host's file system.
- A debugging session for a specific rank is now forcefully exited once the endpoint finishes execution. This contains the debugging experience within user-authored code. It is also necessary for preventing hangs, because if pdb is allowed to continue indefinitely, then control flow will eventually bubble back up to the main asyncio event loop on the worker, at which point everything breaks.
- Hitting a breakpoint now automatically enables post-mortem debugging, so any rank that encounters an exception after hitting a breakpoint will automatically stop at the exception. Attaching the debugger to that rank should then provide an experience like `pdb.post_mortem()`.

## Next steps/gaps I'm aware of (reviewers please read):
- Indexing debug sessions by rank isn't sustainable, because two actor meshes may simultaneously hit breakpoints on the same rank and cause a collision inside the debug client.
- Entering the debug client should happen automatically, rather than requiring the user to do `await debug_client().enter.call_one()`.
- Casting pdb commands should ideally leverage `MeshTrait` rather than reimplementing the selection logic.
- If a mesh was reshaped/renamed so that its dimension names aren't `hosts` and `gpus` anymore, the debugger should reflect the new shape/names.
- The user should be able to enable post-mortem debugging without having to hit a separate breakpoint first.

Differential Revision: D77568423

Reviewed By: zdevito
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D78047960

@facebook-github-bot facebook-github-bot added CLA Signed This label is managed by the Meta Open Source bot. fb-exported labels Jul 10, 2025
samlurye pushed a commit to samlurye/monarch-1 that referenced this pull request Jul 10, 2025
Summary:

Previously, debug sessions were keyed by rank, but this doesn't allow for multiple actors on the same rank. This diff updates the debugging logic so that it is keyed on actor name + rank instead of just rank. The `attach` and `cast` commands now require the user to specify actor name in addition to rank. The `list` command now sorts breakpoints by grouping all actors with the same name (i.e., from the same actor mesh) together, and then sorts by rank within each group.

Differential Revision: D78047960
samlurye pushed a commit to samlurye/monarch-1 that referenced this pull request Jul 10, 2025
Summary:

Previously, debug sessions were keyed by rank, but this doesn't allow for multiple actors on the same rank. This diff updates the debugging logic so that it is keyed on actor name + rank instead of just rank. The `attach` and `cast` commands now require the user to specify actor name in addition to rank. The `list` command now sorts breakpoints by grouping all actors with the same name (i.e., from the same actor mesh) together, and then sorts by rank within each group.

Differential Revision: D78047960
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D78047960

samlurye pushed a commit to samlurye/monarch-1 that referenced this pull request Jul 10, 2025
Summary:
Pull Request resolved: pytorch-labs#494

Previously, debug sessions were keyed by rank, but this doesn't allow for multiple actors on the same rank. This diff updates the debugging logic so that it is keyed on actor name + rank instead of just rank. The `attach` and `cast` commands now require the user to specify actor name in addition to rank. The `list` command now sorts breakpoints by grouping all actors with the same name (i.e., from the same actor mesh) together, and then sorts by rank within each group.

Differential Revision: D78047960
Summary:
Pull Request resolved: pytorch-labs#494

Previously, debug sessions were keyed by rank, but this doesn't allow for multiple actors on the same rank. This diff updates the debugging logic so that it is keyed on actor name + rank instead of just rank. The `attach` and `cast` commands now require the user to specify actor name in addition to rank. The `list` command now sorts breakpoints by grouping all actors with the same name (i.e., from the same actor mesh) together, and then sorts by rank within each group.

Differential Revision: D78047960
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D78047960

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Meta Open Source bot. fb-exported
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants